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Abstract — This  paper  considers  the  search  for  targets  mod¬ 
eled  as  a  discrete  state,  continuous-time  Markov  process. 
Convergence  properties  are  analyzed  using  the  eigenvalues 
and  eigenvectors  of  a  state  transition  rate  matrix  without 
explicitly  solving  differential  equations  or  calculating  matrix 
exponentials.  It  also  studies  the  effect  of  cueing  on  convergence 
rate  using  eigenvalue  analysis  and  optimal  control  theoretic 
approach. 

I.  INTRODUCTION 

In  any  system-of-systems  analysis,  consideration  of  de¬ 
pendencies  between  systems  is  imperative.  In  this  paper, 
we  consider  a  particular  type  of  system  interaction,  called 
cueing.  The  interaction  could  be  between  similar  systems, 
such  as  two  or  more  wide  area  search  munitions,  or  be¬ 
tween  dissimilar  systems,  such  as  a  reconnaissance  asset 
and  a  munition.  In  this  paper,  we  consider  two  types  of 
search  vehicles  that  communicate  target  information  to  other 
vehicles,  an  operation  we  call  cueing.  In  Shakespeare’s 
day,  the  word  cue  meant  “a  signal  (a  word,  phrase,  or 
bit  of  stage  business)  to  a  performer  to  begin  a  specific 
speech  or  action.”  The  word  is  now  used  more  generally  for 
anything  serving  a  comparable  purpose.  Here  we  mean  any 
information  that  provides  focus  to  a  search;  e.g.,  information’ 
that  limits  the  search  area  or  provides  a  search  heading. 

Search  theory  is  one  of  the  oldest  areas  of  operations 
research  [1],  with  a  solid  foundation  in  mathematics,  proba¬ 
bility  and  experimental  physics.  Yet,  search  theory  is  clearly 
of  more  than  academic  interest.  At  times,  a  search  can 
become  an  international  priority,  as  in  the  1966  search  for 
the  hydrogen  bomb  lost  in  the  Mediterranean  Sea  near  Palo- 
mares,  Spain  involving  34  ships,  2,200  sailors,  130  frogmen 
and  four  mini-subs.  The  search  took  75  days,  but  might  have 
concluded  much  earlier  if  cueing  had  been  utilized  from  the 
start.  A  Spanish  fisherman  had  come  forward  quickly  to  say 
he  had  seen  something  fall  that  looked  like  a  bomb,  but 
experts  ignored  him.  Instead,  they  focused  on  four  possible 
trajectories  calculated  by  a  computer,  but  for  weeks  found 
only  airplane  pieces.  Finally,  the  fisherman  was  summoned 
back.  He  sent  searchers  in  the  right  direction,  and  a  sub 
located  the  bomb  under  2,162  feet  of  water  [2], 

Koopman  pioneered  the  application  of  mathematical 
process  to  military  search  problems  during  World  War  II  [1], 
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He  discusses  the  case  in  which  a  searcher  inadvertently 
provides  information  to  the  target,  perhaps  allowing  the 
target  to  employ  evasive  action  [3],  The  use  of  receivers 
on  German  U-boats  to  detect  search  radar  signals  in  World 
War  II  is  a  classic  example.  Koopman  referred  to  this  type 
of  cueing  as  “target  alerting.” 

Ablavsky  et  al.  [4]  proposed  a  geometric  approach  for 
constructing  an  optimal  path  to  search  for  a  moving  target  in 
an  arbitrarily  shaped  region.  The  path  construction  method 
seeks  to  minimize  total  flight  time  and  to  maximize  the 
likelihood  of  finding  the  target.  However,  the  method  re¬ 
quires  an  initial  target  location,  a- terrain  map  that  specifies 
the  target’s  mobility  over  a  region,  and  a  set  of  predefined 
shapes  for  which  optimal  coverage  patterns  are  known. 

The  problem  of  cooperative  search  for  stationary  targets 
using  multiple  search  vehicles  was  investigated  by  Baum  et 
al.  [5].  The  search  environment  is  partitioned  into  ceils,  and 
a  detection  function  is  defined  which  relates  the  probability 
of  detecting  a  target  in  a  particular  cell  to  the  effort 
expended  in  that  cell  (e.g.,  time  spent  searching  the  cell). 
The  classical  “effort  allocation  algorithm”  proposed  by 
Stone  [1]  is  used  to  define  an  optimal  search  plan  without 
considering  practical  constraints  on  vehicle  trajectory.  The 
optimal  plan  is  then  modified  to  construct  a  physically 
feasible  trajectory.  The  modified  plan  is  evaluated  through 
simulation  by  comparing  the  total  search  effort  required  by 
the  modified  plan  with  the  effort  required  by  the  optimal 
plan.  The  evaluation  is  based  on  a  priori  probability  maps 
for  target  locations,  with  actual  target  locations  fixed  and 
initial  search  vehicle  locations  and  headings  varied  over  all 
simulated  cases.  Cooperation  is  modeled  by  allowing  search 
vehicles  to  share  information  concerning  where  each  vehicle 
has  searched  and  whether  or  not  a  target  was  found. 

Curtis  etal.  [6]  proposed  a  coordination  strategy  to  simul¬ 
taneously  accomplish  area  search  and  task  assignment.  The 
problem  consists  of  multiple  homogeneous  search  agents 
and  multiple  fixed  targets  with  unknown  initial  conditions. 
Each  target  has  both  a  position  in  the  search  space  (an 
undirected  graph)  and  a  value.  Search  agents  have  partial 
knowledge  of  the  position  and  value  of  known  targets 
and  the  routes  of  other  agents.  Dynamic  constraints  are 
modeled  by  linear  constraints  on  the  time  required  for  an 
agent  to  reach  a  point  in  the  search  region.  The  objective 
function  measures  the  total  value  of  unengaged  targets;  this 
objective  is  used  both  to  measure  the  value  of  searching  a 
region  and  the  value  of  engaging  a  target.  Individual  and 
team  objectives  are  traded  whenever  an  individual  agent  re¬ 
plans  its  own  route  or  chooses  to  engage  a  target  using  an 
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objective  function  common  to  all  agents,  but  based  on  that 
agent’s  partial  knowledge  of  past  actions  of  other  agents. 

Slater  [7]  examined  the  benefits  of  cooperation  for  two 
search-and-engage  vehicles  searching  for  targets  in  a  region 
containing  both  real  and  false  targets.  Vehicles  cooperate  by 
providing  to  each  other  the  location  of  any  target  attacked. 
The  unengaged  vehicle  will  move  to  the  attacked  target, 
complete  a  damage  assessment,  and  attack  the  target  if 
the  first  vehicle  did  not  destroy  it.  If  the  target  is  already 
destroyed,  the  second  vehicle  will  continue  its  own  search. 
Real  and  false  targets  are  assumed  randomly  distributed 
according  to  a  uniform  distribution  over  the  search  area, 
with  the  probability  of  encountering  targets  following  a 
Poisson  distribution  with  the  Poisson  parameter  based  on 
the  target  density.  Slater  compared  the  number  of  targets 
destroyed  with  and  without  cooperation  for  various  model 
parameters,  such  as  target  density  and  probability  of  correct 
target  identification.  v  ' 

This  paper  investigates  the  effect  of  cooperation  on 
the  search  performance  analytically.  We  use  a  Markov 
chain  analysis  to  examine  cueing  as  a  coupling  mechanism 
between  searchers.  A  Markov  chain  approach  to  target 
detection  can  be  found  in  [I],  which  deals  with  the  optimal 
allocation  of  effort  to  detect  a  target.  A  prior  distribution  of 
the  target’s  location  is  assumed  known  to  the  searcher.  The 
states  correspond  to  cells  that  contain  a  target  at  a  discrete 
time  with  a  specified  probability.  In  this  research,  the  states 
correspond  to  detection  states  for  individual  search-and- 
engage  vehicles.  Jeffcoat  [10]  examined  the  case  of  two 
cooperative  searchers.  He  determined  the  effect  of  cueing 
on  the  probability  of  target  detection  via  a  Markov  chain" 
analysis  and  uses  matrix  exponentiation  to  obtain  numerical 
solutions.  He  showed  the  effect  of  cueing  on  convergence 
for  a  two  vehicle  example.  This  paper  analyzes  the  effect 
of  cueing  on  convergence  more  generally  by  examining 
convergence  properties  of  continuous-time  Markov  chains. 
The  approach  used  in  this  paper  is  analogous  to  convergence 
properties  of  discrete-time  Markov  chains  in  [11],  It  utilizes 
the  eigenvalues  and  eigenvectors  of  a  state  transition  rate 
matrix  without  calculating  computationally  intensive  matrix 
exponentials  or  solving  a  set  of  differential  equations. 

II.  PROBLEM  DESCRIPTION 

Consider  a  search-only  vehicle  with  a  detection  rate  of 
A  targets  detected  per  unit  of  time,  and  a  search-and- 
engage  vehicle  with  a  detection  rate  of  9  detections  per  time 
unit.  Assume  that  cueing  increases  the  search-and-engage 
vehicle’s  detection  capability  by  a  factor  of  k.  That  is,  let 
the  cued  detection  rate  for  an  individual  search-and-engage 
vehicle  be  given  by  k6  per  time  unit.  We  assume  that  once 
a  search-only  or  search-and-engage  asset  detects  a  target, 
it  immediately  cues  one  of  the  search-and-engage  vehicles. 
This  cue  could  take  the  form  of  a  target  coordinate,  a  search 
heading,  or  any  other  information  that  improves  detection 
rate.  We  wish  to  examine  the  impact  of  these  coupled 


system  capabilities  (A,  9,  and  k)  on  the  time  required  to 
engage  targets. 

The  detection  rate  can  be  considered  a  measure  of  some 
system  characteristic  (e.g.,  sensor  capability)  or  of  the 
conflict  scenario  (e.g.,  target  density).  The  cueing  factor,  k, 
represents  the  value  of  the  information  communicated.  The 
higher  the  value  of  k ,  the  more  valuable  the  information, 
where  value  might  represent  the  precision  of  a  target  coordi¬ 
nate  or  the  timeliness  of  the  information  communicated.  We 
assume  that  there  are  sufficient  targets  in  the  battle  space 
so  that  all  search-and-engage  vehicles  have  the  opportunity 
to  find  and  engage  at  least  one  target.  As  a  measure  of 
effectiveness  (MOE),  we  use  the  time  required  to  achieve 
a  threshold  probability  that  all  search-and-engage  vehicles 
have  engaged  a  target. 

III.  STATE  MODEL 

This  section  briefly  describes  continuous-time  Markov 
chains  and  modeling  of  vehicle  states  using  continuous-time 
Markov  chains. 

A.  Continuous-Time  Markov  Chains 

A  continuous-time  Markov  chain  is  a  Markov  process 
x(f)  consisting  of  a  family  of  staircase  functions  with 
continuities  at  the  random  points  tn  [12],  that' is,  transitions 
between  the  discrete  states  occur  at  any  time.  We  denote  by 

Pi(t)  =  P{x(f)  =  aj 
the  state  probabilities  of  x(f)  and  by 

*y(ti»*2)  =  P{x(i2)  =  aj\x(ti)  =  di} 

its  transient  probabilities.  These  functions  satisfy  the  fol¬ 
lowings: 

y  j  ftjj  (^i>  ^2) =  i 
j 

=  Pjfa)- 

i 

A  Markov  process  x(f)  is  homogeneous  if  its  transition 
probabilities  depend  on  the  difference  r  =  f2  —  ti 

it ij(r)  -  P{x(£  +  r)  =  aj\x{r)  =  aj. 

A  Markov  matrix  n(f),  or  a  state  transition  matrix,  is 
defined  by  a  matrix  whose  ij  element  is  jry(i).  We  denote 
by  • 


9ii  ■ ' 

■  <?ln 

Ini 

Qnn 

the  state  transition  rate  matrix  whose  elements  = 
7fy(0+)  are  the  derivatives  from  the  right  of  the  elements 
7 tij(r).  Note  that  V  =  0  because  7r y  (r)  =  1.  Also 

we  can  say  qu  =  -  <hj  <  0  and  >  0,  i  ^  j. 

Since  p{t  +  r)  -  \pi(t  +  r)  •••  pn(t  +  t)}  =  p(t)Il{t), 
we  obtain 


p(t)  =  p(t)Q 


and  its  formal  solution  is 

p(t)  =  p0eQt 

where  p0  —  p(0). 

B.  Modeling  of  Vehicle  States 

We  model  the  possible  vehicle  states  using  discrete  states, 
with  transitions  between  states  possible  at  any  real-valued 
time  greater  than  zero.  For  example,  the  search-only  vehicle 
has  two  possible  states  —  search  and  detect  and 
cue.  Note  that  the  search-only  vehicle  transitions  from  the 
search  state  to  the  detect  and  cue  state  at  rate  A, 
and  that  the  transition  back  to  the  search  state  is  instan¬ 
taneous,  denoted  by  an  infinite  transition  rate.  That  is,  once 
the  search  vehicle  cues  a  search-and-engage  vehicle,  the 
search  vehicle  immediately  resumes  its  search  for  additional 
targets.  The  system  model  is  a  continuous-time  Markov 
chain  because  transitions  between  the  discrete  states  can 
occur  at  any  time. 

Search-and-engage  vehicles  have  three  possible  states, 
search  uncued,  search  cued,  and  detect  and 
engage.  The  transition  rate  from  the  uncued  to  the  cued 
state  depends  on  the  detection  rate  of  the  search-only 
vehicle.  We  assume  that  the  search-only  vehicle  will  cue 
only  one  search-and-engage  vehicle  for  each  target  detected, 
and  that  the  cues  are  equally  distributed  if  all  search-and- 
engage  vehicles  are  available.  Under  these  assumptions,  the 
transition  rate  from  search  uncued  to  search  cued 
is  A/n  if  n  search-and-engage  vehicles  are  available,  but 
increases  to  A  if  only  one  search-and-engage  vehicle  is 
available  to  accept  a  cue.  This  assumption  implies  tha^  the. 
search  vehicle  is  aware  of  the  current  state  of  both  search- 
and-engage  vehicles,  and  the  search  vehicle  can  transmit 
information  to  a  single  vehicle.  Even  if  a  transmission  is 
broadcast  on  a  common  frequency,  we  assume  that  the 
transmitted  data  can  be  tagged  for  use  only  by  an  individual 
search-and-engage  vehicle. 

Recall  that  search-and-engage  vehicles  have  the  ability 
to  search  independently,  so  that  a  vehicle  could  transi¬ 
tion  directly  from  the  search  uncued  to  the  detect 
and  engage  state.  We  also  assume  that  each  search-and- 
engage  vehicle  is  a  single  shot  asset,  so  that  detect  and 
engage  is  a  terminal  state.  Since  the  state  transitions  of 
search-and-engage  vehicles  are  unidirectional,  e.g.,  there 
is  no  transition  to  search  uncued  from  each  state 
and  no  transition  to  search  cued  from  detect  and 
engage,  the  state  transition  rate  matrix  has  triangular  form. 

Example  1 

Let  us  consider  two  search-and-engage  vehicles.  Table  I 
shows  the  states  of  two  vehicles,  in  which  “U”  denotes 
search  uncued,  “C”  denotes  search  cued,  and  “D” 
denotes  detect  and  engage.  The  state  diagram  of  the 
system  is  shown  in  Figure  1.  Then  we  can  formulate  the 
transition  rate  matrix  Q  €  R9x9  with  an  upper  triangular 
form.  □ 


TABLE  I 

State  space  for  two  search-and-engage  (S&E)  vehicles. 


State 

S&E  1 

S&E  2 

Next  State 

! 

U 

U 

2.3,4.7 

2 

U 

c 

3,5,8 

3 

U 

D 

6.9 

4 

C 

U 

5,6,7 

5 

C 

c 

6,8 

6 

c 

D 

9 

7 

D 

U 

8.9 

s 

D 

C 

9 

9 

D 

D 

Fig.  1.  System  state  diagram  for  two  search-and-engage  vehicles 


IV.  MAIN  RESULTS 

In  this  section,  we  provide  convergence  properties  of 
continuous-time  Markov  chains  without  explicitly  solving 
differential  equations  or  calculating  matrix  exponentials.  We 
derive  a  bound  on  settling  time  for  the  state  probabilities. 
We  also  examine  the  characteristics  of  the  transition  rate 
matrix  associated  with  continuous-time  Markov  chains.  Fi¬ 
nally,  we  apply  these  results  to  analyze  the  effect  of  cueing 
in  a  search  application.  The  search  problem  is  modeled 
using  a  linear  quadratic  regulator  (LQR)  formulation. 

A.  Convergence  Rate  of  Continuous -Tone  Markov  Chains 
Proposition  1  Any  transition  rate  matrix  Q  has  an  eigen¬ 
value  equal  to  0. 

Proof:  Define  the  vector  v  =  [1  1  •  •  •  1]T,  then  it  is 

easily  verified  that  Qv  —  0  from  the  fact  that  qy  =  0 
for  all  1  <  i  <  n.  ■ 

The  next  theorem  provides  an  upper  bound  on  the  differ¬ 
ence  between  the  transient  state  value  and  its  steady  state 
value. 

Theorem  1  Suppose  that  Q  £  CnXn  can  be  transformed 
to  the  Jordan  form  Q  =  < Hag  ( Jlt  •  •  •  ,  Jm)  where  the  size 
of  the  Jordan  block  J,  is  jit  i  =  1 ,  ■  ■  ■  ,  m  and  has  non¬ 
positive  eigenvalues  Ai  =  0,  A2,  -  •  •  ,  Am  corresponding  to 


Jordan  block  Ji,  ■  •  •  ,  Jm,  respectively.  Assume  that  ji  —  1. 
Then, 


m  ji  ji  —  k+1 


.k-1 


IIpW  -  pH  E  £  I]  ll^fc+i-ill eX<t  (%  l\)\ 

i=2  k= 1  tel  Ki¬ 

rn 

where  p  =  limbec  p(£).  Vi,vi+ 1,  •  •  •  ,  are  general¬ 

ized  left  eigenvectors  corresponding  to  A*,  i  =  1,  •  •  •  ,m, 
and  ak,  fc  =  1,  •  •  •  ,n,  are  the  complex  coefficients  satisfy¬ 
ing  po  =  Efc=i  afc^- 

Proof:  Let  us  define  T  =  [vj  ■■  ■  vf]T.  Using  that 
pit)  =  poeQt  and  TQ  =  QT,  we  have 


p(t)  =  ^2  akv^Qt 


k=  1 


fc= 1 
n 


—  ^ (  /  +  (Jf  H-  -^-£2  H - 


=  £  (afct,*T_1  (^+<3*+|r*2 

=  [ai  an]e^T. 

Observing  that 


e  Jti  =  cAit 


'1  t  t2/ 2! 
0  1  t 


tji  V(ii  - 1)! 
“  2)! 


0  0  0 


=  eXitMi, 


we  have 


p(£)  =  [oi  ■  •  •  a„]  diag  (eAl*Mi,  •  •  •  ,eXmtMm )  T 


m  ji  ji-k+1  fc_1 

=  EeAi*E  E 

i=l  fc=l  tel  \  )■ 

Since  A,  <  0,  i  =  2, ,m,  we  have  eXittk~l/{k  - 
1)!  — +  0  for  a  finite  k  as  t  — *  oo,  thus,  p  =  ai^i.  Therefore, 
we  have 


IIp(£)  —  pH  = 


m  ji  ji-k+l 

£‘A'*E  £  aeVk+e-i 

i=2  fc=l  tel 
m  ji  ji-k+1 


t 


■k-1 


sEE  E  ||a*vfc+<_1||  eXitj^— 

i=2  fc=l  tel  '  ^  ~ 


(*  -  1)! 
tk~l 

(k-  1)! 


vv/iere  p  =  limt_0Op(£),  Ui ,  •  ■  •  ,  on  are  a  6a.si.5-  of  left 
eigenvectors  corresponding  to  \\  —  0,  Ao, • • ■  ,  A„  respec¬ 
tively,  and  ak,  k  =  1,  •  •  •  ,  n,  are  the  complex  coefficients 
satisfying  p0  =  Eitei  akVk- 

Proof:  It  can  be  easily  proved  by  setting  m  =  n  and  ji  =  1, 
i  =  1,  •  •  •  ,n  in  Eq.  (1).  •  ■ 

The  next  example  illustrates  how  eigenvalues  and  eigen¬ 
vectors  can  be  used  to  estimate  an  upper  bound  of  conver¬ 
gence  of  Markov  chains. 

Example  2 

Consider  the  case  with  two  search-and-engage  vehicles 
described  in  the  previous  section  with  A  =  0.1,  8  =  0.05, 
k  =  4  and  p0  =  [1  0  0  0  0  0  0  0  0],  The  transition 
rate  matrix  Q  is  shown  in  Eq.  (5).  Computing  eigenvalues 
and  generalized  eigenvectors,  we  have  the  Jordan  form 
Q  =  TQT~l.  See  Eq.  (6)  and  Eq.  (7).  They  were  computed 
by  using  MATLAB  function  jordan. 

Note  that  there  is  only  one  Jordan  block  with  size  2. 
From  Theorem  1,  we  observe- that  ||p(f)  —  p||  <  0.2  when 
t  >  30.42.  If  we  consider  the  ninth  state  only,  we  have 
Ipg  (i)  —  1|  <  0.2  when  t  >  29.53.  See  Fig.  2.  The 
solid  (top)  line  denotes  the  bound  of  ||p(f)  -  p||,  that 

is.  Ete2  Eti  Eteifc+1  \\atVk+e-i\\  ex'Hk~'/(k- 1)!,  the 

dash-dotted  (middle)  line  the  bound  of  |p9 (i)  —  1|,  and  the 
dashed  (bottom)  line  the  true  trajectory  of  1  —  pg{t).  □ 


If  the  matrix  Q  is  diagonalizable,  Theorem  1  reduces  to 
the  following  corollary. 

Corollary  1  Suppose  that  Q  €  Cnxn  is  diagonalizable  and 
has  non-positive  eigenvalues,  then 

n 

IIp(£)  -p||  <  E  lla*ufcll eXkt  (2) 

k=2 


B.  Analysis  of  Properties  of  p{t) 

This  section  examines  the  relationship  between  the  state 
transition  rate  matrix  Q  and  convergence  properties  of  p(t). 
The  following  proposition  states  under  which  conditions 
p(f)  has  one  absorbing  state. 

Proposition  2  Suppose  that  Q  has  an  eigenvalue  0  with 
algebraic  multiplicity  1  and  the  other-- eigenvalues  are 
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negative  with  arbitrary  multiplicities.  Then,  if  Q  is  a  trian¬ 
gular  matrix,  p(t)  has  only  one  absorbing  state,  viz.,  p{t) 
converges  to  one  state  with  probability  1  as  t  — >  oo. 

Proof:  Since  each  off-diagonal  element  of  Q  is  nonnega¬ 
tive,  the  row  with  diagonal  element  0  has  all  zero  elements, 
that  is,  if  qkk  —  0,  qki  =  0,  V  i  =  1,  •  •  •  ,  n.  Without  loss 
of  generality,  we  assume  k  —  n.  Then,  the  left  eigenvector 
associated  to  the  eigenvalue  0  is  v\  =  [0  0  •  •  •  0  1], 
Since  p  —  aiv i,  p  has  only  one  non-zero  element.  f 

s 

The  next  proposition  provides  conditions  for  the  non¬ 
absorbing  cases. 

Proposition  3  Suppose  that  Q  has  an  eigenvalue  0  with 
algebraic  multiplicity  m  >  1  and  the  other  eigenvalues  are 
negative  with  arbitrary  multiplicities.  If  Q  is  a  triangular 
matrix  and  the  size  of  Jordan  block  Ji,  i  =  1,  •  •  •  ,m,  is 
one,  then  p(t)  does  not  have  an  absorbing  state  and  p  has 
m  non-zero  elements. 

Proof:  Following  the  proof  of  Proposition  2,  p  = 
YlT=i  aivi  and  p  has  m  non-zero  elements.  ■ 

C.  Analysis  of  Cueing  Effect 

Since  we  have  assumed  that  cueing  increases  the  search- 
and-engage  vehicle’s  detection  capability  by  a  factor  of  k, 
we  can  expect  a  large  value  of  k  increases  convergence 
rate.  First,  let  us  consider  the  case  when  Q  is  triangular. 
In  this  case,  it  is  easy  to  obtain  the  eigenvalues  of  Q  since 
they  are  the  diagonal  elements  of  Q.  It  can  be  noticed  from 
Theorem  1  that  the  state  probabilities  converge  faster  when 
the  eigenvalues  are  more  negative.  This  is  illustrated  in  the 
following  example. 


Fig.  3.  Effect  of  Cueing 


Example  3 

Consider  the  case  with  two  se_arch-and-engage  vehicles  de¬ 
scribed  in  the  previous  section.  The  eigenvalues  dependent 
on  k  are  —  (k  +  1)0  —  A,  —2 k6  and  —  kd.  As  k  increases, 
they  become  more  negative  since  9  >  0.  The  trajectories  of 
P9(f)  are  plotted  in  Figure  3  for  k  —  1  (solid  line)  ,  k  =  4 
(dashed  line),  k  =  10  (dash-dot  line),  and  k  =  30  (dotted 
line).  We  observe  that  ps(t)  converges  faster  as  the  value 
of  k  increases.  □ 


Proposition  4  Suppose  that  all  eigenvalues  of  Q  are  non¬ 
positive  and  Ai(fc),  •  •  •  ,  A m(k)  are  the  eigenvalues  ofQ(k) 
dependent  upon  k.  The  Markov  chain  with  transition  rate 
matrix  Q{k\)  converges  faster  than  one  with  Qfkf)  if 
^t(ki)  <  ^1(^2),  ■  •  ■  .  Am( k\)  <  Ai (At2 )- 

Proof:  Omitted.  >  i  ■ 
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Let  us  consider  non-triangular  cases.  Since  we  assume 
that  cueing  increases  an  individual  searcher’s  detection 
capability  by  a  factor  of  k,  it  can  be  noticed  that  the  state 
transition  rate  matrix  Q  is  affine  with  respect  to  k,  i.e., 
Q  =  kQi  +  <32-  If  we  define  x(t)  —  p(t)T,  we  have 

x{t )  =  QTx(t)  —  4-  kQ^x(t). 

The  system  can  be  transformed  into  a  state  feedback  con¬ 
figuration 

x(t )  =  Ax(t )  -f  Bu(t) 
u(t )  =  —kx(t) 

where  A  =  Qj  and  B  =  —Qf.  This  problem  can  be 
considered  as  a  linear-quadratic  regulator  (LQR)  problem 
with  cost  function 

•  ^ 

T  % 

V  =  f  (x{t)tWx{t)  +  u(t)tRu(t))  dr 
Jo 

+  x{T)t  Mx{T)  (3) 

It  is  well-known  (see  [13])  that  if  (A,  B)  is  controllable  the 
controller  minimizing  the  above  cost  function  is  given  by 

k  =  R-lBTP(t)  (4) 

where  P(t)  is  the  solution  of  the  following  matrix  Riccati 
equation 

-P  =  AtP  +  PA  +  W  -  PBR-lBTP 
P(T)  =  M.  " 

In  order  to  observe  the  effect  of  cueing,  define  W  =  0 
and  M  =  pi.  If  we  make  the  value  p  large,  the  cost  function 
V  penalizes  the  state  x  at  time  T  and  makes  x(T)  small. 
When  we  consider  a  constant  k,  we  can  observe  a  large 
k  penalizes  the  state  x  at  time  T  more,  thus,  makes  fast 
response  of  the  system. 

V.  CONCLUSION 

We  have  considered  a  cooperative  search  for  targets 
with  communication  of  cues,  or  target  information,  between 
searchers.  The  search  is  modeled  as  a  discrete  state,  continu¬ 
ous  time  Markov  process.  We  have  investigated  convergence 
properties  of  continuous-time  Markov  chains  and  provided 
a  bound  on  settling  time  for  the  state  probabilities.  The 
bound  is  a  function  of  eigenvalues  and  eigenvectors  of 
the  state  transition  rate  matrix  and  provides  information 
on  convergence  rate  or  response  time  without  explicitly 
solving  a  set  of  differential  equations  or  calculating  a 
matrix  exponential.  Finally,  we  have  applied  these  results 
to  analyze  the  effect  of  cueing  in  a  search  application. 
We  have  also  provided  a  linear  quadratic  regulator  (LQR) 
formulation  for  the  search  problem. 
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